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ABSTRACT 



An earlier review by the author reported an 



aptitude-treatment interaction (ATI) relating instructional feedback and 
prior knowledge. For low prior knowledge learners, single- try feedback (STF) 
was slightly superior to multiple-try feedback (MTF) , while for high prior 
knowledge learners, MTF was superior to STF. In the current report, raw score 
data from existing studies that examined STF and MTF, but that had not 
considered the possibility of an ATI, are combined and reanalyzed by ANOVA 
(ANalysis Of VAriance) . This study addresses the following questions: is 
there an interaction between learner prior knowledge and number of feedback 
tries provided by computer-based training (CBT) lessons; and, if so, how 
large is this effect? The results of this analysis are presented along 'With a 
tentative explanation of the observed effect. Prescriptive application of the 
ATI and suggestions for future study are provided. (Author/MES) 
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A review by Clariana (1993) reported an aptitude-treatment interaction (ATI) relating instructional 
feedback and prior knowledge. For low prior knowledge learners, single-try feedback (STF) was slightly 
^ superior to multiple-try feedback (MTF), while for high prior knowledge learners, MTF was superior to 

^ STF. In this present report, raw score data from existing studies that examined STF and MTF, but that had 

not considered the possibility of an ATI were combined and reanalyzed by ANOVA. The results of this 
^ analysis are presented along with a tentative explanation of the observed effect. Prescriptive application 

Q of the ATI and suggestions for future study are provided. 

W 



Relevance 



Frequently in corporate settings and occasionally in general education settings, mastery of lower level skills 
are often critical to safety or performance. For example, consider the multitude of alarm sounds that can occur in a 
nuclear power plant and the correct employee reactions to these alarms. This message includes verbal information, 
lower level intellectual skills, and possibly some attitudinal objectives, and it would be hard to dispute that this 
particular message is not important to the employees safety and also their feelings of well-being. In this case, 
computer-based training (CBT) can accurately produce the correct sounds and can require the employee to describe 
the nature of the emergency and the correct responses to each category of emergency (believe it or not, it's not 
always run away as fast as you can). This type of CBT module would be easy and relatively inexpensive to develop 
and deliver (especially compared to instructor led training), and allows for new employee training just-in-time and 
refresher training for existing employees as needed due to the infrequency of such alarms. Decades of educational 
research support the use of practice with feedback for learning this type of information. This presentation argues that 
learners' prior knowledge should be considered when deciding which type of feedback to use in the lesson design. 
This information has immediate practical application for CBT designers. 

Note that this is not new data, but is a reana lysis of four previous experimental studies and an expansion of 
an earlier review' by Clariana (1993). Richey (1998) has described these two processes as an accepted form of 
replicability, though caution must be taken that this present report is not seen as new data. This reanalysis and 
expansion seem acceptable because of the limited circulation of that earlier review due to the demise of the Journal 
of Computer-based Instruction at that time and the lack of information regarding this possible ATI when the four 
experimental studies were originally conducted. Further, the audience for this presentation are the most potent group 
for utilizing this information. 



Introduction 

Advances in voice recognition technology will soon allow relatively accurate evaluation of spoken words, 
and neural network software technology may eventually allow evaluation of sentences and paragraphs. However, 
currently computers are limited in the types of learner responses that can be evaluated in order to give corrective ’ 
feedback. Computers can evaluate learner recognition tasks like clicking on an image or a word, and some free 
recall tasks, such as typing in words and short phrases. 

Feedback in these circumstances has consistently been shown to be better than no feedback (for exan^le, 
see Clariana, Ross, & Morrison, 1991). But is one type of feedback superior to another? A meta-analysis by 
Bangert-D towns, Kulik, Kulik, and Morgan (1991) reported effect sizes (ES) that compared no feedback to (a) 
right/wrong feedback, usually referred to as knowledge of response feedback (KR), (b) knowledge of correct 
response feedback (KCR), (c) try-again feedback (requires multiple tries), and (d) elaborative feedback. Compared 
to no feedback, KR was negatively effective (ES = -0.08), followed next by KCR feedback, ES = 0.22. Repeat until 
correct and elaborative feedback were equally most effective (ES = 0.53). The performance results for feedback may 
be summarized as: KR < no feedback < KCR < Repeat until correct = Elaborative. Prescriptively, elaborative 
feedback and multiple-try feedback were equally most effective, however elaborative feedback typically is more 
difficult and costly to design. 

A review of 30 studies by Clariana (1993) took this a step further by comparing forms of multiple-try 
feedback (MTF) to no feedback, KR feedback, KCR feedback, and delayed feedback. The median effect size for 
MTF compared to no feedback based on 16 effect size measures from 12 studies was ES = 0.56. This finding is 
nearly identical to the results of Bangert-Drowns et. al. (1991) above. However, unlike that meta-analysis, the 
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results for MTF compared to KR, KCR, and delayed feedback were all mixed, with median effect sizes close to no 
difference. 

To explain these mixed findings, Clariana (1993) suggested that low-prior knowledge learners benefited 
slightly from single-try feedback forms relative to multiple-try feedback forms, while high-prior knowledge learners 
benefited greatly from multiple-try feedback forms relative to single-try feedback forms. If supported, this provides 
CBT designers with clear guidelines for selecting number of feedback tries based on audience analysis data and/or 
ongoing lesson performance. 

The current presentation considers this feedback by prior knowledge aptitude treatment interaction (ATI) by 
combining and reanalyzing data from the existing studies described in Clariana (1993). This study addresses the 
following questions: (a) Is there an interaction between learner prior knowledge and number of feedback tries 
provided by CBT lessons? (b) If so, how large is this effect? 

Approach 

A review of the literature was conducted to identify all available studies that compared MTF and STF while 
also providing raw score data. It was necessary' that besides posttest measures, the data set must also contain some 
measure of prior knowledge of the content, so that individual subjects could be classified as "low'" or "high" in terms 
of prior knowledge. In all cases, lesson data ultimately was used to form low and high prior knowledge groupings 
based on median split of lesson scores. 

■ Three dissertaiions and one published report were obtained that met these criteria. All four utilized computer-based delivery 
of the instructional materials. The studies included: Noonan (1984) n = 60, Clariana and Smith (1989) n = 48, Nielson 
(1990) n = 173, and Clariana (1990) n = 40. 

■ Posttesi raw scores were convened to standard scores within each study, then these standard scores were combined into one 
data set and analyzed by 2x2x4 analysis of variance (ANOVA) with the factors feedback (MTF and STF), prior knowledge 
(low and high), and study (the four studies listed above). The final sample contained n = 321 subjects. 

The main effect for number of feedback was not significant (see Table I). The main effect for prior 
knowledge was highly significant F(I, 305) = 24.21, p=.000I. As would be expected, those students that performed 
well on the CBT lesson also performed better on the posttest. This finding is interesting in that in experimental 
studies using grouping by lesson performance, there is typically a regression to the mean. The “low” group tends to 
gain quite a bit since their lesson scores are low, while the “high” group gains less and may even decline. If the 
experimental treatment is not strong enough, this regression to the mean causes a leveling effect that can wash-out 
low and high group differences. 

Table I. Posttest scores Analysis of Variance 



Source 


df 




MS 


F 


S 


Feedback tries (T) 


1 


0.14 


0.14 


0.16 


0.69 


Prior knowledge (K) 


1 


22.02 


22.02 


24.21 


0.00 


FxT 


1 


5.05 


5.05 


5.55 


0.02* 


Study (S) 


3 


0.06 


0.02 


0.02 


0.99 


TxS 


3 


2.15 


0.72 


0.79 


0.50 


Kx S 


3 


0.45 


0.15 


0.16 


0.92 


TxKxS 


3 


1.96 


0.65 


0.72 


0.54 


Error 


305 


277.34 


0.91 







A significant interaction of feedback and prior knowledge was observed F(I, 305) = 5.55, p=.02. For low 
prior knowledge students, single-try feedback was more effective than multiple-try feedback, ES = O.I I (see Figure 
I). For high prior knowledge learners, multiple-try feedback was better, ES = 0.39. 
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Prior Knowtedge 

The low-prior knowledge students made large lesson (L) to posttest (P) gains with both single-try and 
multiple-try feedback (see Figure 2). The high-prior knowledge students, since they had already scored high on the 
lesson measure of content knowledge, did not make gains as great as the low-prior knowledge students. In addition, 
the high prior-knowledge students receiving multiple-try feedback (the right vector of the “high” pair of vectors) 
made larger gains on the posttest relative to the high-prior students receiving single-try feedback (the left vector of 
the “high” pair of vectors) even though both groups already had good content knowledge. 




Discussion 



prior knowledge 



To summarize, this reanalysis supports an ATI for prior knowledge and number of feedback tries. Our 
tentative explanation of this effect involves an attitudinal impact on learning. Note that for these four studies, and 
for most if not all MTF studies so far, there is no difference in presentation form between STF and MTF when the 
learner's response is correct. The difference in presentation form between STF and MTF occurs only with lesson 
errors, in that the learner must respond again with MTF but is usually provided the correct answer without additional 
responses with STF. Dick and Latta (1970) reported that low ability students became frustrated with repeat until 
correct (MTF) recall lessons relative to high able students and also the STF treatment. To examine possible 
frustration with MTF, we recently collected item-by-item posttest responses for twenty of our students who were 
taking a computer-based module as an assignment in an instructional design class. In that module, it was possible to 
intentionally skip posttest questions (recall level) by entering either nothing or else entering nonsense letters 
(“garbage”) and pressing Enter. Figure 3 below shows the total number of intentionally skipped posttest items for 
the STF group (n=10) and MTF group (n=10). Note that the jx)sttest was identical for each treatment, thus the 
pattern observed relates to attitude carried over from the preceding lesson activity. 
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Figure 3. Cumulative posttest *^skip ” or "garbage ” responses. 




posiiesi question number 

This data leads us to feel that, as in the Dick and Latta (1970) study, students may experience some 
frustration with MTF, and this will likely negatively impact learning. 

At the same time, in a survey of the same class, 71% of the students said that they would prefer to try-again 
(MTF) if given the choice. Some specific comments included: ”I learn more when I get it on my own” and ’Trying a 
second time makes me think more about the question and what it is really asking”. This suggests that these learners 
and maybe learners in general have a bias towards "getting it right on their own”, and so if they don't miss so many 
items that it becomes punishing, then they actively engage the items and learn more (relative), thus accounting for the 
higher performance for MTF with high prior knowledge learners. But if the learner misses a number of items, they 
begin to become frustrated and may eventually stop trying to learn, thus the lower performance for MTF (relative) 
for the low prior knowledge learners. Providing the correct answer after one try (STF) tends to reduce frustration for 
the students who miss a lot of items, but it seems to come at the cost of reduced engagement for the students who are 
doing well. Thus, the type of feedback given can influence the learners approach to the lesson, and on-going lesson 
performance serves as advisement allowing the student to decide to engage in or disengage from the lesson. 

Requiring high-prior knowledge learners to try again in CBT has intuitive appeal, and can be implemented 
by tracking lesson performance and basing feedback prescription of STF or MTF on that performance. A lesson may 
start out with MTF, if a students on-going lesson performance drops below some designated value, say 80%, then the 
lesson changes to STF. 

Does this finding generalize to other, and non-CBT settings? For example, in a constructivist-based 
classroom, what are the ramifications of asking questions to high-prior knowledge students versus the low-prior 
knowledge students. Asking a difficult question and requiring a student to develop the answer parallels the MTF 
condition with low prior knowledge students. This approach may be frustrating to the student, and could result in 
disengagement and reduced learning. Of course, teachers in class have additional options to mitigate frustration, but 
the question remains about what is the best approach in terms of questioning level and which students should be 
asked to respond. If this ATI holds for classroom instruction, in-class questions should be carefully crafted and 
targeted to the individual student, along the lines of Vygotsky's zone of proximal development (Vygotsky, 1986), and 
maybe asking some "easy” questions is a good idea. 

Though this ATI is only supported by four experimental studies, there is enough evidence to warrant future 
studies to confirm or deny it, and possibly to expand on these findings to other settings. Though it is not cuirently 
fashionable to investigate "lower level” domains such as verbal learning using simple (or not so simple) recognition 
tasks like answering multiple-choice tests, nevertheless, these domains are an important and foundational to learning; 
and multiple-choice items, especially with feedback, are one tool among many that any instruction designer may use. 
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